Method and system of displaying information during a medical procedure

ABSTRACT

A method and system including a head mounted display for displaying information to a user when performing a medical procedure is described. A user wearing the head mounted display device can view objects, while simultaneously receiving and displaying feedback information on a procedure being performed. Text and graphical information is presented in a position on the head mounted display so that the object or a part of an object out of the line of sight appears in the same location and with the same shape, size and orientation as if the object or part of an object were visible to the user.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a method and system of displayinginformation during a medical procedure in which a head mounted displayis used to display information to the user.

Description of Related Art

Intramuscular injections may damage peripheral nerves at the site ofinjection. The site of injection is the crucial factor in determiningthe degree of nerve fiber injury. During medical procedures, nurses anddoctors lack real time accurate information about where certain nervesor organs are in relation to a syringe needle or other medical devicesthat they are using during the procedure.

Surgical operations may inadvertently damage organs due to a surgeon nothaving real time information about the location of an organ, and thedistance between surgical tools and the organs since the surgeon reliesonly on his experience and memory.

For patients in isolated locations, remotely located surgeons may assistlocal personnel in performing difficult surgical operations. The localpersonnel may not know exactly what he is looking for.

SUMMARY OF THE INVENTION

The present invention relates to a method and system including a headmounted display for displaying information to a user when performing amedical procedure. Preferably the head mounted device is at leastpartially see-through. A user wearing the head mounted display devicecan view objects, while simultaneously receiving and displaying feedbackinformation on a procedure being performed, including living organs andtools within the user's field of view. For example, the feedbackinformation can be text information. The text is presented at a positionon the head mounted display so that the information does not interferewith the user's ability to perform or assist in the procedure. Thefeedback information can also include graphical information representingan object or a part of an object out of the line of sight. The graphicalinformation is presented in a position on the head mounted display sothat the object or a part of an object out of the line of sight appearsin the same location and with the same shape, size and orientation as ifthe object or part of an object were visible to the user so that itseems to the user that the graphical representation is a real object.The displayed parts of objects are a simulation and not the real object.As such, the displayed object would be only an approximation of the realobject.

In a further aspect, a computer implemented method provides theinformation to the head mounted display to enhance the ability of a userwearing the head mounted display to perform a medical procedure. Themethod can be initiated by receiving an indication that a user isperforming a medical procedure. The user's field of view through thehead mounted display is determined and objects within the field of viewboth in the line of sight and out of line of sight at the sight of themedical procedure. Relevant objects for the medical procedure areselected and information about the relevant objects that are in thefield of view of the user are retrieved. The relevant object'sinformation can include the shape, size location or the orientation.Supplemental information which is to be provided to the user can becalculated or retrieved from data storage. The supplemental informationis displayed in the head mounted display and can change as the user'sfield of view changes or relevant object's properties and actions withinthe medical procedure change.

The invention will be more fully described by reference to the followingdrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a schematic diagram of a user with a head mounted displayused while performing a medical procedure (an injection).

FIG. 1B is a schematic diagram of a display of information directed tothe medical procedure (an injection) on the head mounted display shownin FIG. 1A.

FIG. 2 is a schematic diagram of a system for displaying informationduring a medical procedure.

FIG. 3A is a flow diagram of a thread of execution for processing userhead and eyes movement.

FIG. 3B is a flow diagram of a thread of execution for trackingproperties of objects such as location, shape, size and orientation.

FIG. 3C is a flow diagram of a thread of execution for combining boththe user head and eyes information and the objects' location,orientation and shape, and determining how and where to display theresulting information, and then displaying the information on the headmounted display.

FIG. 4 is a flow diagram of an alternate thread of execution fortracking the objects location, shape and orientation, in which a humanoperator is responsible for updating objects' properties, so that theycorrespond to the ones from the medical procedure.

FIG. 5 is a flow diagram of main application thread for execution of themethod of the present invention.

DETAILED DESCRIPTION

Reference will now be made in greater detail to a preferred embodimentof the invention, an example of which is illustrated in the accompanyingdrawings. Wherever possible, the same reference numerals will be usedthroughout the drawings and the description to refer to the same or likeparts.

FIGS. 1A and 1B shows user 100 wearing a head mounted display device 103performing a medical procedure. As used herein a medical procedure is aprocedure performed on an animal, such as a human being, performance ofa practical work, or an action or operation performed for diagnosing aproblem or disease. The medical procedure, in this example, is aninjection.

As used herein an object is anything within the medical procedure,including people, animals, living organs, tools or other objects, bothin the line of sight or out of the line of sight. All the objects andactions within the field of view of a user comprise a scene. Scenesgenerally change over time.

User 100 has a field of view 131 as shown in FIG. 1A. Within field ofview 131 are various objects, such as living bodies, living organs,and/or medical tools. Some of these objects may be partially or totallyout of the line of sight, and in effect invisible to the user.

As used herein attending the medical procedure means either performingthe medical procedure or assisting somebody else performing the medicalprocedure. As used herein the field of view is the angular extent of agiven scene that is imaged by a camera or seen by eyes.

As used herein a database is a collection of data organized especiallyfor rapid search and retrieval. As such, both relational database and aflat file database are databases.

During the medical procedure, the user may change his location in theroom where the medical procedure is performed or may move their head oreyes. The present invention utilizes various techniques to track theuser's location, the user's head and eyes location and orientation todetermine the user's field of view, in order to present the user withsupplemental information regarding the operation at which the user is inattendance.

The main application and thread for performing the method for displayinginformation during a medical procedure start at step 512, as shown inFIG. 5. The user logs in at steps 512 and 516. In step 516, it isdetermined if the user is authorized. If the user is authorized,information display application 232 receives an indication that the useris attending a medical procedure as shown in FIG. 2. Referring to FIG.5, step 520 is performed to activate secondary threads described in FIG.2, 3A, 3B, 3C or 4.

As shown in FIG. 1A, the user viewing an medical procedure would see theoperation subject and tools. User 100 is wearing head mounted displaydevice 103. Head mounted display device 103 is transparent orsemitransparent, with the capability to overlay information in displayfield 135. Head mounted display device 103 is capable of generatingdisplay elements on various portions of head mounted display device 103while remaining portions of head mounted display device 103 aretransparent to allow the user to continue to view objects during thecourse of the medical procedure. Head mounted display device 103 iscapable of generating display elements on various portions of a user'sdisplay so that it gives the user the illusion of 3D representations ofportions of objects that are out of line of sight. Visible objects 101are in field of view 131 and in the line of sight.

As used herein a displayed object is an object out of the line of sightwhose graphical representation is generated on the display.

As used herein, a model of an object is information that is sufficientto be used by a head mounted display device to display an approximationof the object at the same position, size and shape as though the objectwas visible to the user. As used herein, a generic model of an object isinformation that describes the object but it is not sufficient to beused by a head mounted display device to display an approximation of theobject with the same position, size and shape as though the object wasvisible to the user. As used herein, integrating is used for obtaining amodel of an object. Information must be integrated for each head mounteddisplay device individually. A generic model of an object can beintegrated with information from sensors to obtain a model of thedisplayed object. When displaying the model of the displayed object, themodel of the displayed object appears to the user at the same locationand with the same shape, size and orientation as if the displayed objectwere visible to the user. The precision of the displayed model providesuseful feedback information to the user.

FIG. 1B illustrates supplemental information which can be provided touser 100. FIG. 1B is a partial view of display field 135 of one lens ofa head mounted display device 103 such as that shown in FIG. 1A. In FIG.1B, supplemental information display elements 107, 111, 115, 118, 121,125 are displayed on display field. Supplemental information displayelement 107 including text, supplemental information display element 115including a bulls eye and supplemental information display element 125including text are intended to teach or assist the user. Supplementalinformation display element 111 including an out of sight element,supplemental information display element 118 including a graphic displayobject of the sciatic nerve and supplemental information element 121 areintended to give feedback to the user. Supplemental information 118displays graphically a representation of the sciatic nerve. Supplementalinformation display element 111 displays graphically a representation ofthe used tool in the medical procedure, for example a syringe needle.Both the sciatic nerve and at least portions of the syringe needle arenot in the line of sight, and hence, not visible. Supplementalinformation display element 121 includes text of a calculated distancebetween the displayed object of the sciatic nerve and the displayedobject of a needle of syringe 129.

FIG. 2 is a block diagram of system for displaying information during amedical procedure 200. Head mounted display device 103 is coupled todisplay processor 254. Head mounted display device 103 includes a numberof sensors 298. Sensors 298 can include cameras, orientation sensors,location sensors and other sensors. Display processor includes networkinterface 256, processor 258, memory 260, microphone input 266 locationsensors input 268, and/or orientation sensors input 270. Displayprocessor 254 is coupled to head mounted display device 103 through anynumber of various means, as described below. Display processor 254 isconnected to network 252, such as the Internet, using network interface256 to couple head mounted display device 103 to supplementalinformation provider 201. Processor 258 executes programmaticinstructions from application 262 and other services described herein.Display processor 254 can include any of the examples of processingdevices described herein. Camera input 264 and microphone input 266provide respective visual and sound information to supplementalinformation provider 201 and application 262. Camera input 264 canreceive image and video data from sensors 298 of head mounted displaydevice 103 which sensors 298 may include one or more of a face forwardcamera, one or more eye facing camera, or both. Microphone input 266 canreceive audio from a microphone on the head mounted display device 103or on a processing device associated with the user and the display.Orientation sensors input 270 receive input from one or moreaccelerometers, gyroscopes and magnetometers on head mounted displaydevice 103, the data from which can be used to measure the head locationand orientation of the user. Other sensors inputs 274 receiveinformation from additional sensors from user sensors 298.

Supplemental information provider 201 includes network interface 238,processor 236, memory 204, object orientation sensors 244, objectlocation sensor 242, other object sensor 246, object informationdatabase 248 which can containing generic models of at least some of theoperation objects and operation information database 250. Both objectinformation database 244 and operation information database 250 can beflat file databases. Other embodiments can use other types of databases,such as relational databases, or use general data storage. Supplementalinformation provider 201 is connected to network 252, such as theInternet, using the network interface 238. Processor 236 executesprogrammatic instructions from application 232 and other servicesdescribed herein. Supplemental information provider 201 can includestorage means for supplemental operation information 206, storage meansfor the information for identifying relevant objects 210, storage meansfor user data including location and orientation 218, informationdisplay application 232, image recognition application 226 andauthorization application 234. Operation and object data 206 storesinformation about the objects from object information database 248integrated with data retrieved from object orientation sensors 244,object location sensors 242, other object sensors 246 for exampleultrasound sensors or other sensors. Information about the medicalprocedure relevant objects and instructional supplemental informationare stored in operation information database 250.

The information for identifying relevant objects can be a unique id or aunique name for each relevant object. The information for identifyingrelevant objects can contain unique type ids or unique type names aswell. In this case, each syringe type and each needle type can have anunique id, and each individual syringe and each individual syringeneedle can have their own id.

As an example, information for identifying relevant objects is retrievedfrom the operation information database 250 and saved to the informationfor identifying relevant objects memory location 210. Information aboutthe syringe 129, syringe needle and the sciatic nerve from FIG. 1B areretrieved from object information database 248 using the information foridentifying relevant objects. Then relevant object properties which caninclude location, orientation, shape are retrieved using object locationsensors 242, object orientation sensors 244 and other object sensors246, again using the information for identifying relevant objects. Theinformation retrieved from operation information database 250 and theinformation retrieved from object location sensors 242, objectorientation sensors 244 and other object sensors 246 are integrated byinformation display application 232. Thereafter, supplementalinformation display element 111 such as the needle can be displayed. Ageneral shape and location for the sciatic nerve is retrieved fromobject information database 248, and then the sciatic nerve shape,orientation, size and location is updated using information from cameras240 and using image recognition application 226 at step 416 in FIG. 4.Thereafter, supplemental information display element 118 can bedisplayed, or supplemental information display element 121 can becalculated and displayed as shown in FIG. 1B.

Supplemental information elements 111 and 118 can be displayed in 3D ifthe user chooses that the supplemental information be displayed for botheyes at the same time. Alternatively, the supplemental information canbe displayed for only one eye, so that the user is not overwhelmed withinformation.

FIG. 3A, FIG. 3B and FIG. 3C represent flow diagrams of parts of themethod of the present invention. There are three separate threads. Thesethreads are part of the information display application 232 shown inFIG. 2. The memory locations accessed by multiple threads are protectedby locks. Other embodiments can use processes instead of threads, andmutexes, critical sections or other mutual exclusion mechanisms insteadof locks. Other embodiments can use any combination of threads,processes, or polling and the corresponding mutual exclusion mechanisms.

The thread of execution for processing user head and eye movementsstarts at step 304. The user's movements are monitored at step 304. Theuser movements' monitoring includes information being retrieved fromlocation sensors 268 and orientation sensors 270. The retrievedinformation is checked for change such as movements that at step 308. Ifuser movement was determined, a new field of view is calculated at step312. Lock for user data 222 shown in FIG. 2. is locked at step 316, andthen the head and eye information and the field of view are stored atstep 320. The head and eye information and the field of view are storedat operation data memory location 218 shown in FIG. 2. The lock for userdata 222 shown in FIG. 2 is unlocked at step 324.

The thread for execution for tracking properties of starts at step 337as shown in FIG. 3B. The information for identifying relevant objects isretrieved at step 337. The relevant objects' properties which caninclude movement or shape are monitored at step 341. The relevantobjects' properties monitoring includes retrieving information that caninclude the shape, size, location or orientation of the relevant objectsthat are in the field of view of the user at the moment. The retrievedinformation is checked for changes at step 343. Information about theobjects is retrieved from the object information database 248 shown inFIG. 2 at step 345. Object information is integrated at step 347 usingboth data retrieved from the database and data retrieved from thesensors at step 341, for example to create models of displayed objectsof the medical procedure. Lock for operation and object data 220 shownin FIG. 2 is locked at step 351. Thereafter, the integrated objectinformation, such as models of the operation objects, is stored at step355. The integrated object information is stored at the operation andobject data memory location 206 shown in FIG. 2. Lock for operation andobject data 220 shown in FIG. 2 is unlocked at step 359.

The thread of execution for displaying the information starts at step370 as shown in FIG. 3C. Lock for operation and object data 220 shown inFIG. 2 is locked at step 370, and stored integrated information such ashead and eye information and the field of view are loaded at step 374.The integrated object information is loaded from operation and objectdata memory location 206 shown in FIG. 2. Lock for operation and objectdata 220 shown in FIG. 2 is unlocked at step 378. Interactions ordistance between objects are calculated at step 380. Lock for user data222 shown in FIG. 2. is locked at step 382, and then stored userinformation such as head and eye information and the field of view areloaded at step 386. Lock for user data 222 shown in FIG. 2. is unlockedat step 390. Determining which elements of supplemental information fromleast the relevant objects in the field of view and out of line of sightis performed at step 394. Data, shapes and text, colors, text size todisplay are determined at step 394 as well. For example, users canselect between different types and amounts of supplemental informationto be displayed, such as for teaching or feedback supplementalinformation, size of the text, types of graphical representation for thedisplayed objects such as wire-frame colors for the displayed objects,text and other preferences. The head mounted display is updated at step398.

As an example of integrating object information at step 347, the lengthof the syringe needle and information about the syringe (a genericmodel) is retrieved from object information database 248, the locationand the orientation of the syringe, information about how full a syringeis and the type of the syringe and the syringe needle is recovered usingcamera inputs 264 from the cameras of sensors 298 and image recognitionalgorithms or additional sensors. The information retrieved from thedatabase represents a model of a syringe and a model of a needle. All ofthe information is used to completely determine the location,orientation, size and shape of the syringe and the syringe needle evenif the syringe is partially or totally out of line of sight to create amodel of the syringe and of the syringe needle that approximate the realobjects.

Another example of integrating object information at step 347, theshape, orientation and location for an organ (sciatic nerve in our case)for a generic person (a generic model of the sciatic nerve) is retrievedfrom object information database 248. The information retrieved from thedatabase represents a model of the organ. Then the size and location ofthe actual patient are retrieved using additional sensors 246. Theinformation retrieved from the database and the information retrievedfrom the sensors are integrated and the shape, location, orientation andsize of the organ are obtained. The new shape, orientation and locationrepresents a model that approximates the shape, orientation and locationof the real organ.

At step 394, display processor 254 calculates points and polygons on asurface of the syringe and the syringe needle using informationcalculated at step 347 and the user's head and eye information whichincludes the field of view retrieved at step 304, and loaded at step386. Display processor 254 determines what information and where theinformation should be displayed to the user, and displays parts of thesyringe and the syringe needle that are both in the field of view andout of the field of sight such as supplemental information displayelement 111, supplemental information display element 118 such as anorgan, supplemental information display element 107 such as supplementaltext information, supplemental information display element 115 such as abulls eye and supplemental information display elements 121 and 125 suchas supplemental text information shown in FIG. 1B.

The supplementary information to be displayed to the user at step 398can include: shapes of the relevant objects in the field of view and outof line of sight of the user, instructional supplemental informationsuch as the place of an injection or an incision, or other supplementalinformation such as the distance between the syringe needle and thesciatic nerve. A warning can be displayed as text or graphics if thesyringe needle and the sciatic nerve get too close to each other.

FIG. 4 illustrates an alternate embodiment for the thread for executionfor tracking properties of starts at step 337. The information foridentifying relevant objects is retrieved at step 337. The humanoperator input which can include the relevant objects' movement or shapechanges are monitored at step 408. The retrieved information is checkedat step 343. Information about the objects is retrieved from database230 shown in FIG. 2 at step 345. Object information is integrated atstep 416 using both data retrieved from the database and data retrievedfrom the user input at step 408. Lock for operation and object data 220shown in FIG. 2 is locked at step 351. Thereafter, the integrated objectinformation is stored at step 355. The integrated object information isstored at the operation and object data memory location 206 shown inFIG. 2. The lock for operation and object data 220 shown in FIG. 2 isunlocked at step 359.

In another embodiment, object sensors 246, such as ultrasound, canretrieve real time information such as shape, orientation, locationabout the relevant objects. Information from the object informationdatabase 248 can help identify the relevant objects and isolate them.Processor 236 will run information display application 232 for isolatinginformation about the relevant objects and recovering the shape of therelevant objects. For example, the isolated information about therelevant organs represents the models of the organs. For the ultrasoundsensor example, information display application 232 can use forisolating objects rules that are usually used for reading an ultrasoundimage, such as skin appears smooth and bright, fat can be bright ordark, but subcutaneous fat is typically dark, muscle is also dark, whenviewed in cross section, but in long section sound is reflected back bythe muscle fibers and the internal structure of the muscle can be easilyseen, tendons are typically bright, but this varies with theirorientation relative to the probe, bone appears as a particularly brightline bright line due to the dramatic difference in acoustic impedancebetween bone and soft tissue. Operation and object data 206 stores themodels of the relevant objects. The precision should be good enough sothat the displayed model gives useful feedback information to the user.In this embodiment, models of organs such as the heart can be displayedmoving in real time with the same shape, size and orientation as if theobject were visible to the user.

In another embodiment, sensors 246, such as ultrasound, retrieve 3D realtime information about some of the objects from the scene. Thereafter,this information can be displayed as transparent or semitransparentvoxels making up objects with the same shape, size and orientation as ifthe user was able to watch the real objects being transparent orsemitransparent. The colors of the voxels are either the ones that camedirectly from the sensor, or are alternative colors. Processor 236 willrun the information display application 232 for calculating the size andthe position of the voxels and alternative colors.

In another embodiment, sensors 246, such as ultrasound, can retrieve 2Dreal time information about some of the objects from the scene. Ineffect the 2D real time information represents a section through theobjects. Then this information will be displayed as a projection on aplane as though the user was able to watch a section through the organs.The colors of the texture pixels are either the ones that came directlyfrom the sensor, or are alternative colors. Processor 236 will runinformation display application 232 for calculating the orientation andthe position of the plane and the texture to be projected onto the saidplane.

The present embodiment uses one main calculator containing processor 236and memory 204 and a secondary calculator containing processor 258 andmemory 260 for each user. Other embodiments can use differentconfigurations, such as only one calculator, containing processor 236and memory 204, and also performing the work performed by the secondarycalculators.

It is to be understood that the above-described embodiments areillustrative of only a few of the many possible specific embodiments,which can represent applications of the principles of the invention.Numerous and varied other arrangements can be readily devised inaccordance with these principles by those skilled in the art withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A method for displaying information to a userduring a medical procedure comprising the steps of: a. determining afield of view of the user through a head mounted display device worn bythe user using one or more sensors for determining the field of view,the one or more sensors for determining the field of view include activesensors for sensing head orientation of the user, head location of theuser, eye or eyes location of the user, or the eye or eyes orientationof the user; b. retrieving object information on one or more objectswithin the medical procedure within the field of view from the one ormore sensors, the one or more objects within the medical procedureinclude one or more medical tools used in real time in the medicalprocedure and living bodies and/or living organs of the medicalprocedure, checking if the properties of the one or more object changed,the properties being selected from location, shape, size andorientation; c. retrieving by a processor supplemental objectinformation from data storage, said supplemental object informationincludes one or more 3D generic models of the one or more objects fromstep b. in the field of view; d. integrating by a processor saidretrieved object information on one or more objects within the field ofview from the one or more sensors from step b. with the retrievedsupplemental object information from step c. in real time to obtainmodels of the objects that can be displayed as a graphicalrepresentation with the same shape, size and orientation as through thesaid objects were visible to the user; and e. displaying by a displayprocessor the models of objects of step d. in 3D on one or more displayelements on the head mounted display device in real time, the objectsbeing in the field of view, and repeating steps a. through e. for eachmovement of the user which changes said field of view of the user andevery time said one or more objects changes a property selected from oneor more of a location, orientation or shape, wherein said head mounteddisplay is transparent or semi-transparent to allow the integratedinformation to be overlaid on the field of view to allow the user tocontinue to view objects of the one or more medical tools used in realtime in the medical procedure and the living bodies and/or living organsin the field of view during the medical procedure.
 2. The method ofclaim 1 further comprising the step of: calculating relationshipsbetween objects using the integrated information obtained in step d. 3.The method of claim 2 wherein the relationship is a distance between theone or more objects of the object information and the one or moreobjects of the supplemental information.
 4. The method of claim 1wherein said object information is selected from one or more of thegroup of a shape, size, location and orientation from the one or moresensors.
 5. A system for displaying information to a user comprising: ahead mounted display device including one or more sensors fordetermining a field of view of the user through the head mounted displaydevice the one or more sensors including active sensors for sensing headorientation of the user, head location of the user, eye or eyes locationof the user, or the eye or eyes orientation of the user; one or moresensors for retrieving object information on one or more objects withinthe medical procedure within the field of view, the one or more objectswithin the medical procedure include one or more medical tools used inreal time in the medical procedure and living bodies and/or livingorgans of the medical procedure, checking if the properties of the oneor more object changed, the properties being selected from location,shape, size and orientation; a processor, the processor retrievingsupplemental object information from data storage, said supplementalinformation includes one or more 3D generic models of one or moreobjects from a medical procedure, the supplemental information includesinformation about the one or more objects of the medical procedure thatare in the field of view, the one or more objects from the medicalprocedure include one or more medical tools used in the medicalprocedure in real time and living bodies and/or living organs of themedical procedure; a processor, the processor integrating said retrievedobject information on one or more objects within the field of view fromthe one or more sensors with the retrieved supplemental objectinformation in real time and obtaining models of the said objects thatcan be displayed as a graphical representation with the same shape, sizeand orientation as though the said objects were visible to the user; anda display processor, the display processor displaying in 3D theintegrated information on one or more display elements on the headmounted display device, said head mounted display is transparent orsemi-transparent to allow the integrated information to be overlaid onthe field of view to allow the user to continue to view objects of theone or more medical tools used in real time in the medical procedure andthe living bodies and/or living organs in the field of view during themedical procedure.
 6. The system of claim 5 wherein said supplementalinformation includes information on shape, size, location or orientationfrom the one or more generic models of one or more objects.
 7. Thesystem of claim 5 wherein said object information includes informationon a shape, size, location or orientation from the one or more sensors.8. The system of claim 5 wherein the processor calculates relationshipsbetween said objects using the integrated information.
 9. The system ofclaim 5 wherein the one or more of head orientation sensors and one ormore head location sensors are one or more face forward cameras.
 10. Thesystem of claim 5 wherein the eye or eyes location sensors and/or eye oreyes orientation sensors is one or more eye facing cameras.